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Executive  Summary 


The  Software  Engineering  Institute’s  (SEI)  experience  with  a  variety  of  DoD  acquisition  pro¬ 
grams  has  shown  the  need  for  improvements  in  acquisition  and  sustainment  product  evaluation 
practices.  This  technical  note  proposes  a  comprehensive  approach  to  product  evaluation  that  will 
reduce  acquisition  and  sustainment  costs  by  increasing  the  quality  of  results  and  eliminating  the 
need  for  repeated  testing  and  rework. 

Current  practice  emphasizes  testing  as  the  principal  means  of  determining  whether  a  product 
meets  specified  customer  needs  and  quality  criteria,  but  testing  is  time-consuming,  expensive,  and 
inconclusive.  Testing  inherently  occurs  late  in  the  development  of  a  product  when  discovery  of 
defects  requires  rework;  rework  undermines  the  architectural  integrity  of  the  product  resulting  in 
further  testing,  additional  defects,  and  further  rework. 

Further  experience  in  industry  has  shown  that  a  variety  of  proven  practices  can  augment  and  re¬ 
duce  this  narrow  dependence  on  testing;  these  include  precise  acceptance  criteria,  early  validation 
of  product  requirements  to  those  criteria,  and  a  disciplined  iterative  development  process  for  con¬ 
tinuous  product  integration.  Others  include  more  rigorous,  technically  focused  reviews,  performed 
continuously  throughout  development.  This  will  better  ensure  a  systematic  effort  to  reduce  rework 
through  earlier  detection  of  defects  and  an  emphasis  on  a  product  line  approach  for  eliminating 
redundant  efforts.  Instituting  these  practices  would  provide  the  basis  for  more  efficiently  acquir¬ 
ing  products  that  will  be  of  sound  quality,  responsive  to  the  customer’s  needs,  and  cost-effective 
in  the  face  of  increasingly  constrained  budgets. 
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Abstract 


This  technical  note  focuses  on  software  acquisition  and  development  practices  related  to  the  eval¬ 
uation  of  products  before,  during,  and  after  implementation.  From  engagements  with  numerous 
DoD  acquisition  programs,  it  has  been  observed  that  a  number  of  recurring  issues  reduce  the  ef¬ 
fectiveness  of  how  software-reliant  products  are  evaluated.  An  acquisition  effort  consists  of  iden¬ 
tifying  the  customer’s  needs,  selecting  or  developing  a  product  that  is  responsive  to  those  needs, 
and  then  evaluating  the  product  to  determine  if  it  properly  addresses  the  identified  needs.  This 
technical  note  describes  the  Product  Evaluation  (verification,  validation,  and  certification)  process 
including  test,  reviews,  and  formal  methods.  It  also  makes  the  argument  that  Product  Evaluation 
should  not  be  deferred  until  after  a  product  has  been  built,  but  should  begin  as  soon  as  the  cus¬ 
tomer’s  needs  have  been  identified  and  should  continue  throughout  the  acquisition  effort. 
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1  Introduction 


Acquisition  is  a  systematic  effort  to  obtain  products  whose  capabilities  will  improve  the  ability  of 
an  enterprise  to  perform  its  mission.  An  acquisition  effort  consists  of  identifying  the  customer’s 
needs,  selecting  an  existing  product  or  selecting  a  supplier  to  develop  a  new  product  that  is  res¬ 
ponsive  to  those  needs,  and  then  evaluating  that  the  as-built  product  properly  addresses  the  identi¬ 
fied  needs.  An  acceptable  product  is  deployed  into  use  as  part  of  an  operational  system  and  over 
time  is  sustained  to  continue  satisfying  the  needs  for  which  it  was  built. 

The  effort  of  evaluating  a  product  for  acceptability  should  begin  as  soon  as  the  customer’s  needs 
have  been  identified,  but  not  deferred  until  after  a  product  has  been  built,  and  should  continue 
throughout  the  acquisition.  Evaluation  requires  gaining  and  maintaining  a  proper  understanding  of 
the  customer’s  needs,  advocating  development  practices  that  will  reduce  the  frequency  of  defects, 
and  planning  actions  to  identify  and  diagnose  any  flaws  in  the  product  while  it  is  being  built.  Ear¬ 
ly  detection  of  defects  is  less  costly  both  in  terms  of  the  effort  required  to  make  corrections  and  in 
terms  of  maintaining  product  integrity.  Product  Evaluation  (verification,  validation,  and  certifica¬ 
tion)  encompasses  all  of  the  means  that  can  be  used  to  determine  whether  a  product  is  going  to 
meet  a  customer’s  needs — including  testing,  reviews,  and  formal  methods. 

The  SEI  has  observed  a  number  of  recurring  issues — in  perfonning  engagements  with  numerous 
DoD  acquisition  programs  and  their  associated  development  contractors — concerning  how  soft¬ 
ware-reliant  products  are  evaluated.  This  note  focuses  on  issues  related  to  how  acquisitions  ap¬ 
proach  the  evaluation  of  products  during  and  after  development  and  suggests  various  potential 
improvements  in  these  practices. 

1.1  Observations  on  Common  Practice 

More  than  half  of  a  typical  acquisition  effort  is  spent  evaluating  the  product  that  is  being  obtained. 
The  purpose  of  these  evaluations  is  to  detennine  whether  the  product  is  being  built  properly  and 
whether  it  provides  the  capabilities  that  a  customer  needs  [Boehm  2001,  Bennett  2005]. 

Much  of  the  evaluation  effort  during  product  development  is  indirect,  involving  reviews  of  exten¬ 
sive,  as-built  documentation  without  needed  characterization  of  alternatives  and  tradeoffs  consi¬ 
dered  or  rationale  for  subsequent  decisions.  Without  an  understanding  of  how  these  considerations 
have  influenced  product  development,  evaluation  reduces  to  a  determination  of  consistency  and 
completeness  relative  to  perceived  customer  needs  rather  than  whether  the  product  is  a  best  fit  to 
those  needs  given  development  constraints  of  cost,  schedule,  and  technology. 

After  product  development,  evaluation  reduces  to  anecdotal  testing:  effort  focuses  on  finding  in¬ 
stances  for  which  product  behavior  may  differ  from  expectations  in  representative  cases  of  opera¬ 
tional  usage.  Differences,  when  determined  to  be  defects  in  the  product,  trigger  an  indeterminate 
but  rushed  period  of  rework  during  which  the  product  is  incrementally  revised  to  work  properly 
given  the  chosen  cases,  but  often  also  introducing  new  defects  to  be  found  and  corrected. 

At  the  root  of  this  costly  and  ineffective  approach  to  product  evaluation  is  an  initial  description  of 
the  customer’s  needs  that  is  typically  poorly  organized,  inconsistent,  incomplete,  and  yet  prema- 
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turely  over-specific  in  constraining  potential  solutions.  Uncertainties  about  actual  needs  and  ex¬ 
pectations  for  potential  future  changes  are  seldom  communicated.  Acceptance  criteria  cannot  be 
precisely  articulated  with  such  a  poor  foundation,  but  is  expressed  instead  in  the  form  of  test  cases 
that  reduce  broadly  stated  needs  to  necessarily  narrow  cases  that  are  only  representative  examples 
of  those  needs. 

Reviews  and  testing  will  remain  the  primary  means  that  acquirers  and  developers  use  to  evaluate 
the  acceptability  of  a  product;  however,  we  can  hope  to  improve  these  by  looking  more  closely  at 
what  constitutes  proper  evaluation  of  a  product.  In  doing  this,  we  need  to  consider  several  ques¬ 
tions: 

•  Can  product  evaluation  efforts  be  applied  during  product  development  in  a  way  that  will  sig¬ 
nificantly  reduce  discovery  during  product  acceptance  of  defects  that  cause  delay  and  the 
need  for  costly  rework? 

•  What  is  the  optimum  level  of  product  testing?  How  much  testing  is  needed  to  ensure  that  a 
product  is  acceptable  for  use? 

•  Is  testing  effective  for  finding  all  of  the  defects  that  could  reasonably  be  found?  Are  defects 
being  found  that  should  have  been  found  earlier,  in  reviews  or  prior  testing? 

•  Are  there  techniques  other  than  testing  and  reviews  that  can  be  used  in  order  to  increase  con¬ 
fidence  in  a  product  earlier  or  with  less  effort? 

.  How  can  the  effectiveness  of  work  product  reviews  be  improved  to  ensure  product  quality 
and  reduce  defects  that  would  be  found  later  through  testing? 

•  When  a  product  previously  found  to  be  acceptable  is  modified,  does  the  product  need  to  be 
comprehensively  reevaluated  or  can  the  effort  needed  to  determine  that  the  modified  product 
is  also  acceptable  be  reduced? 

•  In  situations  where  multiple  versions  of  a  product  are  needed — tailored  for  different  uses, 
operational  contexts,  or  technologies — can  evaluations  be  leveraged  in  order  to  minimize  per¬ 
version  evaluation  costs? 

1 .2  The  Nature  of  Product  Evaluation 

There  are  various  views  of  a  product,  each  expressed  through  a  different  representation  [Kopetz 
1976].  These  representations  must  be  mutually  consistent,  with  each  comprising  a  partial  model  of 
the  product  that  answers  only  certain  questions  about  it: 

•  Customer  needs',  expressing  the  customer’s  expectations  concerning  what  capabilities  the 
product  should  provide 

•  Operational  context,  describing  the  physical  and  information  environment  in  which  the  prod¬ 
uct  operates 

•  Acceptance  criteria :  defining  the  customer’s  criteria  (threshold  and  objective)  for  determining 
whether  the  product  is  acceptable  to  be  put  into  use 

.  Requirements',  specifying  the  product’s  expected  behavior 
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•  Design :  specifying  the  architecture,  constituent  components,  and  prescribed  organic  and 
emergent  properties  of  a  product  that  is  a  satisficing  (satisfy  and  suffice)  solution  for  the  re¬ 
quirements 

•  Implementation :  constituting  a  concrete  realization  of  each  of  the  product’s  components 

•  User  documentation',  defining  how  the  product  is  operated  and  used  as  part  of  an  organiza¬ 
tion’s  business/mission  enterprise 

The  relationships  and  differences  among  these  representations  are  illustrated  in  the  following  ex¬ 
ample: 

•  Customer  needs  specify  “no  loss  of  critical  data” 

•  Operational  context  specifies  infonnation  that  is  accessible  to  the  product  from  its  environ¬ 
ment 

•  Acceptance  criteria  specify  what  data  is  critical,  that  any  data  over  30  seconds  old  must  be 
recoverable  within  30  seconds,  and  that  any  resulting  data  gaps  must  be  logged  and  reported 
to  the  system  manager 

•  Requirements  specify  a  product  information  model  (including  critical  data  items)  and  specify 
retention  and  recovery  policies  that  satisfy  critical  data  retention  acceptance  criteria 

•  Design  specifies  a  data  integrity  process  and  components  for  dynamic  and  persistent  storage; 
it  specifies  policies  and  interfaces  for  committing  and  recovering  or  rolling  back  data  to  a 
consistent  state  and  for  detecting  and  handling  data  discrepancies. 

•  Implementation  provides  the  components  that  realize  the  designed  approach  for  ensuring  data 
integrity 

•  User  documentation  describes,  from  a  user’s  perspective,  what  to  expect  in  terms  of  data  cur¬ 
rency  and  what  happens  if  a  gap  occurs 

The  role  of  product  evaluation  in  acquisition  is  to  validate  and  elaborate  the  product  acceptance 
criteria  in  the  form  of  reviews  or  tests  that  are  then  used  to  determine  whether  each  of  the  other 
representations  properly  and  consistently  satisfy  that  criteria.  Through  the  process  of  evaluating 
each  of  the  product’s  representations,  the  product  is  progressively  shown  as  satisfying  the  cus¬ 
tomer’s  needs.  When  discrepancies  arise,  the  cause  must  be  traced  to  its  source  in  one  or  more  of 
the  product’s  representations  and  corrected  to  reestablish  mutual  consistency. 

1.3  Limits  to  Testing  as  a  Product  Evaluation  Method 

With  the  emphasis  given  to  testing  and  the  level  of  effort  expended  on  it  during  an  acquisition,  it 
is  natural  to  assume  that  this  is  the  best,  or  only,  means  that  an  acquirer  can  rely  on  to  be  sure  of 
getting  an  acceptable  product.  Testing  alone,  however,  is  not  an  adequate  technique  for  evaluating 
a  software-reliant  product;  the  limitations  of  testing  are  widely  recognized  and  must  be  effectively 
augmented  with  other  methods  of  evaluation  [Linger  1996;  Spillner  2007]. 

It  is  not  possible  to  determine  conclusively  whether  a  product  has  only  prescribed  behavior  and  is 
free  of  defects.  Testing  can  only  show — in  the  best  case — that  attempted  trials  have  produced  ex¬ 
pected  results  and  have  revealed  no  defects.  The  infinite  combinatorial  variety  of  potential  valid 
and  invalid  inputs  and  internal  data/processing  states  of  a  product  makes  exhaustive  testing  im- 
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possible;  only  a  representative  coverage  of  possible  inputs  and  states  is  feasible  within  a  finite 
schedule  and  budget.  The  most  that  testing  can  achieve  is  a  level  of  confidence  that  testing  per¬ 
formed  is  representative  of  expected  actual  usage  and  that  defects  will  not  arise  in  operational 
uses  that  do  match  the  cases  tested.  This  confidence  can  be  undermined  if  different  users  follow 
different  usage  patterns  that  cannot  all  be  tested. 

Not  only  is  exhaustive  testing  not  feasible,  there  is  not  yet  a  commonly  accepted  objective  meas¬ 
ure  for  determining  when  testing  efforts  have  been  sufficient  to  stop  and  declare  the  product  ac¬ 
ceptable  for  use.  Again,  the  best  option  is  to  determine  what  criteria  will  be  used  to  judge  whether 
testing  has  provided  an  adequate  level  of  confidence  that  all  critical  defects  have  been  discovered. 
The  best  means  for  doing  this  is  to  design  test  sets  that  provide,  based  on  estimated  defect  density, 
a  statistically  sufficient  level  of  code  and  path  coverage,  input  data  boundary  condition  coverage, 
and  scenario-based  usage  coverage. 

A  particular  impediment  to  testing  is  the  necessity  that  an  operable  product  exists  to  be  tested. 
This  means  that  testing  finds  defects  at  the  time  when  rework  is  most  costly.  Whereas  many  types 
of  defects  may  be  discovered  using  other  techniques — such  as  reviews  or  analytic  models  that  are 
applicable  to  incomplete  products,  systemic  defects,  or  emergent  properties  such  as  performance 
or  reliability — defects  can  be  discovered  with  testing  only  through  use  of  the  product  under  spe¬ 
cific  conditions.  These  defects  can  be  a  result  of  fundamental  architectural  decisions  that  require 
significant  effort  to  change. 

In  addition,  many  types  of  processing  logic  are  difficult  to  test,  requiring  initial  computational 
conditions  that  are  difficult  to  replicate.  In  particular,  concurrent  forms  of  processing  (including 
multi-processor  and  multi-threaded,  timing-  and  event-dependent,  and  distributed  latency- 
sensitive  logic)  are  not  understood  well  enough  to  predictably  reproduce  and  diagnose  complex 
defects. 

Testing  must  ensure  not  only  that  the  product  behaves  correctly  under  normal  conditions  but  also 
that  it  behaves  acceptably  in  the  face  of  unexpected  operational  conditions  (including  failures). 
Again,  the  most  feasible  approach  is  to  categorize  potential  unexpected  conditions  and  failures  as 
the  basis  for  test  sets  that  cover  representative  cases. 

Software  operates  within  the  physical  constraints  of  hardware;  results  of  testing  on  different 
hardware  can  produce  different  results  depending  on  the  software’s  sensitivity  to  hardware  cha¬ 
racteristics.  Slow  processors  can  limit  the  software’s  functional  capabilities;  fast  processors  can 
expose  undetected  conflicts  in  concurrent  resource/data  sharing.  Usage  of  needed  hardware  re¬ 
sources  (e.g.,  displays,  data  stores,  sensors,  effectors,  communications)  can  be  restricted  if  other 
software  also  needs  access  to  those  resources  at  the  same  time.  Different  hardware  can  represent 
data  with  different  degrees  of  accuracy,  precision,  or  value  limits.  Results  of  tests  can  change  if 
operational  hardware  differs  from  test  hardware  or  if  any  changes  are  made  in  the  operational 
hardware.  Similarly,  product  behavior,  as  well  as  corresponding  test  results,  can  differ  if  the  be¬ 
haviors  of  interacting  systems  are  inaccurately  modeled  in  testing  or  vary  in  unforeseen  ways. 
Analogously,  training  modes  in  the  product’s  behavior  can  exhibit  these  same  issues  to  the  degree 
that  inaccurate  models  are  substituted  for  inaccessible  parts  of  the  actual  environment. 

The  remainder  of  this  note  presents  a  more  complete  view  of  product  evaluation  as  the  practice 
within  an  acquisition  effort  for  determining  whether  a  product  is  going  to  satisfy  the  stated  needs 
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of  a  customer.  Section  2  gives  an  overview  of  the  elements  of  product  evaluation,  whereas  Sec¬ 
tions  3-5  discuss  (in  more  detail)  the  preparation  for  and  performance  of  evaluations  during  and 
after  product  development.  Section  6  discusses  product  evaluation  in  the  context  of  a  product  line 
acquisition  and  Section  7  suggests  specific  actions  that  can  be  taken  to  improve  product  evalua¬ 
tion  practices  in  general. 
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2  Understanding  Product  Evaluation  in  an  Acquisition 
Context 


Viewed  broadly,  acquisition  encompasses  product  development  and  product  acceptance  (Figure 
1).  Product  development  entails  obtaining  a  usable  product  that  targets  identified  customer  needs, 
whereas  product  acceptance  entails  determining  whether  the  obtained  product  is  a  proper  fit  to 
those  needs.  Product  evaluation  is  an  essential  element  of  both  of  these. 


Needs 

Customer 


Product 

Customer 


Figure  1:  The  Product  Acquisition  Cycle 
More  specifically: 

•  Product  Development  consists  of  the  iteratively  performed  activities  of  systems  and  software 
engineering  (requirements,  design,  implementation,  and  integration)  required  in  creating  a 
product.  The  role  of  product  evaluation  during  development  is  to  verify  that  the  product  is  be¬ 
ing  correctly  developed  (i.e.,  consistently  across  all  work  products  as  specified)  and  that  it  is 
converging  on  specified  customer  acceptance  criteria. 

•  Product  Acceptance  consists  of  integrating  the  product  with  other  system  elements  and  per¬ 
forming  product  evaluation  to  determine  whether  the  product  behaves  correctly  in  its  ex¬ 
pected  usage.  The  product  must  be  verified  against  customer  acceptance  criteria  and  validated 
that  it  provides  the  customer  with  the  operational  capabilities  actually  needed;  any  diver¬ 
gences  from  expectations  must  be  diagnosed  and  characterized  as  defects  for  disposition  in  a 
subsequent  iteration  of  product  development. 

Product  evaluation  begins — at  or  before  the  start  of  product  development — with  the  formulation 
of  customer  acceptance  criteria.  This  representation  is  an  explicit  and  objective  characterization  of 
acquirer-specified  customer  needs  and  gives  product  development  and  product  acceptance  efforts 
a  shared  view  of  those  needs.  Although,  in  the  end,  a  product  will  be  judged  acceptable  only  if  it 
behaves  operationally  in  the  way  that  the  customer  expects,  the  definition  of  formalized  accep¬ 
tance  criteria  helps  to  ensure  that  expected  behavior  has  been  communicated  accurately.  Equiva¬ 
lently,  this  provides  an  early  basis  for  systematically  identifying  and  correcting  any  omissions, 
ambiguities,  or  misunderstandings  concerning  the  customer’s  needs  as  communicated.  With  this 
basis,  product  evaluation  becomes  an  objective  means  for  determining  whether  a  product  should 
be  accepted  for  deployment  and  use. 

During  product  development,  product  evaluation  must  be  performed  both  collaboratively  with 
engineers  and  evaluators  and  independently  to  ensure  a  quality  product  that  will  meet  customer 
acceptance  criteria.  During  product  acceptance,  product  evaluation  should  be  chartered  as  exclu¬ 
sively  representing  the  interests  of  the  acquisition  and  customer  organizations.  Further,  product 
evaluation  should  be  coordinated  across  product  development  and  product  acceptance  to  ensure 
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that  development  and  acceptance  efforts  are  working  toward  satisfying  commonly  understood 
customer  acceptance  criteria  as  the  authoritative  definition  of  targeted  customer  needs.  Develop¬ 
ment  and  acceptance  activities  can  operate  concurrently,  iterating  through  product  versions  until 
the  product  has  been  determined  to  be  acceptable  for  customer  use. 

2.1  Product  Evaluation  Methods 

Although  testing  is  an  essential  element  of  product  evaluation,  it  is  not  the  only  (or  always  best) 
means  of  establishing  the  acceptability  of  a  product  [Linger  1996],  Effective  product  evaluation 
relies  on  a  mix  of  methods  that  can  ensure  early,  least-effort  discovery  of  defects  for  correction. 
These  methods  broadly  include  reviews,  testing,  and  formal  methods,  augmented  by  runtime  me¬ 
thods  that  can  limit  the  impact  of  undetected  defects. 

2.1.1  Reviews 

The  most  cost-effective  technique  for  product  evaluation  is  reviews  by  a  mix  of  mentors,  peers, 
and  subject  matter  experts  [Pomeroy-Huff  2009].  While  testing  requires  an  operable  product  as  its 
subject,  reviews  can  occur  anytime  during  development  as  various  work  products  are  produced. 
By  properly  reviewing  each  work  product,  defects  can  be  discovered  and  corrected  before  work 
has  even  started  on  other  work  products  that  might  depend  upon  it.  Product  evaluators  looking 
across  related  work  products  can  discover  inconsistencies  in  how  developers  understand  the  cus¬ 
tomer’s  needs  and  the  technical  approach  being  taken  and  how  these  inconsistencies  can  be  re¬ 
solved  to  avoid  defects.  To  be  most  effective,  reviews  should  focus  each  reviewer  on  specific 
technical  questions,  particularly  emphasizing  areas  that  the  author  views  as  difficult  or  uncertain. 

2.1.2  Testing 

Testing  is  an  important  means  of  product  evaluation  because  it  builds  directly  upon  knowledge  of 
how  an  enterprise  operates  [Perry  2000;  Bhor  2001;  McGregor  2001;  Spillner  2007].  However, 
testing  is  also  necessarily  an  anecdotal  method  in  that  it  is  impossible  even  to  enumerate,  much 
less  test,  all  of  the  conceivable  ways  that  a  product  might  be  used  under  all  conceivable  condi¬ 
tions.  To  mitigate  the  impossibility  of  exhaustively  testing  a  product  in  all  the  ways  it  may  be 
used,  a  strategy  of  layered  progressive  testing  is  widely  used.  Testing  proceeds  in  layers  corres¬ 
ponding  to  the  logical  structure  of  the  solution,  the  assumption  being  that  testing  at  each  level  will 
be  effective  at  discovering  defects  that  occurred  at  the  corresponding  stage  of  development. 

•  Component:  The  product  design  specifies  a  set  of  components  whose  implementations  com¬ 
prise  the  base  units  from  which  a  product  is  constructed.  Each  component  is  independently 
tested  to  confirm  that  its  implementation  confonns  to  its  design. 

•  Product:  The  product  being  acquired  is  constructed  by  selecting  and  integrating  verified  com¬ 
ponents  in  accordance  with  the  design.  The  product  as  a  whole  is  then  tested,  following  sce¬ 
narios  that  approximate  how  the  product  will  be  used,  to  verify  that  its  behavior  conforms  to 
its  requirements. 

•  System:  The  product  is  integrated  with  other  elements  of  the  customer’s  operational  environ¬ 
ment  to  verify  that  the  product  behaves  as  expected,  as  specified  in  the  customer  acceptance 
criteria. 
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•  Operational:  The  integrated  system  is  validated  as  behaving  according  to  customer  expecta¬ 
tions  in  light  of  current  operational  needs. 

Testing  is  a  means  to  evaluate  a  product  based  on  specific  scenarios  and  test  cases  that  represent 
each  tester’s  understanding  of  how  the  product  will  be  used.  Testers  must  select  scenarios  and 
cases  that  seem  representative  of  the  inputs  and  actions  that  the  product  is  most  likely  to  encoun¬ 
ter  in  actual  use,  as  well  as  cases  that  represent  less  common  or  abnormal  uses  but  that  could  ex¬ 
pose  defects  with  more  significant  detrimental  effects. 

Even  with  a  sound  means  to  identify  cases  that  might  have  been  missed,  testing  alone  is  not  going 
to  discover  every  defect  in  a  product.  The  challenge,  then,  is  determining  how  to  focus  testing 
efforts  to  get  the  best  results  and  how  to  use  other  methods  to  augment  product  quality.  Getting 
better  results  means  having  higher  confidence  that  fewer  significant  defects  remain  to  be  discov¬ 
ered  and  corrected  later.  Significant  defects  are  those  that  have  the  greatest  impact  on  correct  be¬ 
havior  of  the  product,  particularly  those  that  limit  effective  performance  of  the  customer  organiza¬ 
tion’s  mission. 

2.1.3  Formal  Methods 

In  addition  to  reviews  and  testing,  options  are  emerging  for  greater  use  of  fonnal  methods  that  can 
reduce  the  amount  of  time  spent  on  evaluations  or  to  expose  defects  that  are  otherwise  difficult  to 
discover.  Formal  methods  include  techniques  such  as  static  analysis,  static  or  symbolic  testing, 
assurance  cases  and  model-based  dynamic  analyses  for  verifying  quality  factors,  and  domain- 
specific,  correct-by-construction  engineering  methods  [Weinstock  2004;  Feiler  2010]. 

Reviews  and  formal  methods  are  particularly  relevant  in  the  context  of  product  lines  where  testing 
of  individual  products  alone  would  provide  inadequate  leverage.  Reviews  and  formal  methods 
that  can  be  applied  to  an  abstractly  represented  set  of  similar,  yet-to-be-developed  products  pro¬ 
vide  the  means  to  verify  satisfaction  of  criteria  and  expose  defects  across  an  entire  set  of  similar 
products. 

2.1.4  Runtime  Strategy 

Although  a  systematic  effort  to  use  a  variety  of  methods  to  detect  defects  can  improve  the  quality 
of  products,  no  combination  of  methods  can  guarantee  the  complete  absence  of  defects.  To  pro¬ 
vide  a  final  level  of  protection,  there  should  be  a  runtime  strategy  and  mechanisms  implemented 
in  the  product  to  detect  and  handle  failures  due  to  undiscovered  defects,  limiting  their  potential 
impact  and  retaining  information  that  will  enable  subsequent  diagnosis  and  correction.  A  part  of 
product  evaluation  is  to  ensure  that  this  strategy  and  its  associated  mechanisms  are  in  place. 

2.2  Product  Evaluation  Activities  during  Product  Development 

Product  evaluation  during  development  has  the  purpose  of  helping  to  reduce  product  acceptance 
efforts  and  delays  that  result  from  late  discovery  of  defects  requiring  rework.  By  helping  develop¬ 
ers  discover  (understanding  and  construction)  errors  as  early  as  possible,  the  effort  required  to 
correct  those  errors  and  to  maintain  consistency  among  all  work  products  is  minimized. 
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Product  evaluation  activities  during  product  development  consist  of 

•  Reviewing  customer  acceptance  criteria  to  ensure  that  customers’  needs,  as  currently  unders¬ 
tood,  are  being  properly  communicated 

•  Reviewing  requirements  to  ensure  consistency  with  customer  acceptance  criteria  and  other 
available  information  concerning  business  process,  operational  environment,  and  future  cus¬ 
tomer  needs 

•  Reviewing  design  (architecture)  for  consistency  and  completeness  relative  to  requirements 
and  proper  consideration  of  design  alternatives  and  tradeoffs 

•  Reviewing  formal  design-and  implementation-level  models  that  provide  analyses  of  evidence 
for  satisfaction  of  specified  quality  factors 

•  Reviewing  design  (component  specifications)  for  consistency  and  completeness  relative  to 
architecture  and  conventions 

•  Reviewing  internal  designs  of  components  for  consistency  and  completeness  relative  to  their 
specifications 

•  Reviewing  component  implementations  for  consistency  and  completeness  relative  to  their 
specifications  and  internal  designs 

•  Advising  and  assisting  in  the  construction  of  component  and  integration  test  materials  (using 
relevant  elements  of  the  product  testing  infrastructure  as  appropriate)  and  reviewing  compo¬ 
nent  and  integration  test  results  for  evidence  of  defects 

•  Developing  and  performing  integration  tests  to  ensure  that  components  work  together  to  pro¬ 
vide  an  operable  product,  identifying  and  diagnosing  any  discrepancies 

•  Developing  and  performing  product-level  behavioral  testing  to  verify  conformance  with  re¬ 
quirements  (to  identify  and  diagnose  any  behavior  that  diverges  from  expectations) 

•  Analyzing  the  causes  of  any  identified  defect  to  detennine  whether  improvements  in  devel¬ 
opment  or  evaluation  practices  could  avoid  or  detect  such  defects  earlier 

For  increased  effectiveness,  product  evaluators  should  be  collaboratively  paired  with  product  de¬ 
velopers  to  help  identify  and  clarify  any  misconceptions  concerning  customer  needs  and  to  im¬ 
prove  the  quality  of  developer  verification  efforts.  Product  evaluators  that  assist  multiple  develop¬ 
ers  can  help  ensure  consistency  of  understanding  and  solution  approach  among  them.  The  goal  of 
product  evaluation  during  product  development  is  to  discover  and  eliminate  defects  through  the 
active  involvement  of  increased  domain  knowledge  and  continuous  verification.  Although  a  de¬ 
velopment  environment  may  not  permit  exact  replication  of  operational  conditions,  awareness  of 
potential  differences  can  help  to  isolate  the  scope  of  the  impact  of  those  differences  on  the  prod¬ 
uct. 

2.3  Product  Evaluation  Activities  during  Product  Acceptance 

The  purpose  of  product  acceptance  is  to  ensure  that  a  product  will  behave  as  needed  and  expected 
in  the  specified  context  of  its  operational  use.  Product  acceptance  has  two  aspects: 

•  Verification:  determining  that  the  product  is  a  correct  solution  to  the  problem  as  understood 

•  Validation:  determining  that  the  product  properly  addresses  the  customer’s  needs 
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Product  evaluation  for  product  acceptance  occurs  at  two  levels:  system  and  operational.  System 
evaluation  begins  with  system  integration — installing  the  product  with  other  hardware  and  sys¬ 
tems  to  create  a  facsimile  of  the  complete  operational  environment.  The  product  is  then  verified  as 
operating  properly  in  that  environment — following  representative  business  process  scenarios — in 
accordance  with  customer  acceptance  criteria.  Operational  evaluation  validates  whether  the  prod¬ 
uct  will  satisfy  specified  customer  needs  when  used  by  representative  users  in  performing  speci¬ 
fied  business  processes  in  the  actual  (or  a  closely  representative)  business  operational  environ¬ 
ment. 

2.4  Orchestration  of  Product  Evaluation  Activities 

Performing  product  evaluation  requires  more  than  the  direct  efforts  of  reviewing,  analyzing,  and 
testing  the  product  and  constituent  work  products;  it  requires  setting  up  a  framework  that  is  con¬ 
ducive  to  performing  product  evaluations  effectively  and  efficiently.  This  requires  additional  ef¬ 
fort,  entailing  several  activities  that  can  be  performed  prior  to  or  concurrent  with  primary  product 
development  and  product  acceptance  activities: 

•  Augmenting  requirements  derived  from  customer  needs,  by  specifying  required  capabilities 
and  instrumentation  that  will  enable  the  observability  that  is  needed  for  product  evaluation  ac¬ 
tivities 

•  Planning  activities,  staffing,  and  resources  that  are  needed  to  perform  product  acceptance 

•  Building  a  test  environment  by  replicating  or  emulating  the  product’s  specified  operational 
environment 

•  Acquiring  operational  data  and/or  tools  that  can  be  used  to  generate  appropriate  test  data 

•  Acquiring  and  tailoring  the  tools  needed  for  analysis  and  reporting  of  observed  versus  ex¬ 
pected  product  behavior  indicated  by  test  results 

•  Acquiring  and  installing  required  operational  hardware  and/or  emulators  to  support  testing 

•  Development  of  operational  scenarios,  which  consist  of  tests  that  emulate  normal  and  excep¬ 
tional  product  use,  and  definition  for  each  test  of  expected  results,  which  characterize  the 
product’s  expected  behavior 

•  Building  and  installing  the  product — with  appropriate  instrumentation — to  operate  in  the  test 
environment 

•  Performing  tests  to  collect  results  and  operational  profile  data 

•  Analyzing  test  results  and  operational  data  in  order  to  diagnose,  report,  and  track  defects  indi¬ 
cated  by  differences  in  expected  versus  observed  product  behavior 
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3  Prerequisites  to  Product  Evaluation 


The  role  of  product  evaluation  in  acquisition  is  to  determine  whether  a  product  that  is  being  ac¬ 
quired  is  suitable  for  operational  deployment  and  use.  The  objectives  of  this  effort  are  to  ensure 
that  the  purpose  of  the  product  is  properly  understood,  that  corresponding  criteria  for  its  accep¬ 
tance  by  the  customer  are  clear,  and  that  the  product  meets  those  criteria. 

Effective  product  evaluation  begins  from  a  proper  definition  of  the  customer  enterprise  and  the 
activities  of  that  enterprise  that  the  product  is  meant  to  support.  A  proper  definition  of  an  enter¬ 
prise  includes  knowledge  of  the  business  objectives,  roles,  processes,  and  procedures  that  the  en¬ 
terprise  performs.  The  concern  for  product  evaluation  with  this  is  that  the  enterprise  definition 
properly  informs  the  intended  purpose  of  the  product. 

3.1  Acceptance  Criteria  vs.  Requirements 

Acceptance  criteria  and  requirements  are  alternative  representations  of  customer  needs  that  the 
product  is  meant  to  satisfy.  They  differ  in  that  the  acceptance  criteria  define  the  minimal  capabili¬ 
ties  that  the  product  must  exhibit  to  be  acceptable  to  the  customer,  whereas  the  requirements — 
after  refinement  through  analyses  of  engineering  alternatives  and  tradeoffs — characterize  the  ob¬ 
servable  behavior  of  a  specific  solution.  From  a  product  evaluation  perspective,  requirements 
must  specify  product  behavior  that  will  satisfy  customer  acceptance  criteria  and  then  the  product 
as-built  must  satisfy  these  stricter  requirements.  During  product  acceptance,  product  evaluation 
must  only  verify  that  the  product,  which  has  been  shown  during  development  to  meet  validated 
requirements,  as  a  whole  does  in  fact  conform  to  the  acceptance  criteria. 

For  product  evaluation  to  be  effective,  product  development  methods  must  anticipate  the  tech¬ 
niques  that  will  be  used  to  verify  intermediate  work  products.  In  preparation  for  product  develop¬ 
ment,  product  evaluation  should  ensure  that  the  methods  to  be  used,  and  any  supporting  automa¬ 
tion,  provide  for  observability  of  work  product  content  and  include  documentation  of 
assumptions,  alternatives  considered,  and  rationale  for  choices  made. 

A  particular  challenge  in  choosing  development  methods  is  achieving  a  balance  between  describ¬ 
ing  the  capabilities  that  are  needed  in  a  product  and  prescribing  a  specific  solution.  Some  prevail¬ 
ing  techniques  tend  to  promote  concrete,  overly  specific  descriptions  because  these  are  more  easi¬ 
ly  conceived  and  understood  from  the  perspective  of  a  user.  Examples  of  this  are  often  seen  in  the 
common  practice  of  defining  requirements  as  free-form  “shall”  statements  or  as  more  stylized  use 
cases.  These  and  other  such  techniques  benefit  from  a  user’s  ability  to  describe  how  they  work  or 
how  they  expect  a  product  to  behave.  The  drawback  of  these  more  prescriptive  forms  of  specifica¬ 
tion  is  that  they  depend  upon  users  making  assumptions  and  choices  that  prematurely  eliminate 
potentially  better  alternatives.  Furthermore,  the  actual  variety  of  ways  and  conditions  in  which  a 
product  may  be  used  precludes  describing  in  detail  all  of  those  conceivable  uses  and  the  result  is 
that  the  set  of  such  descriptions  is  unavoidably  incomplete;  while  more  abstract  forms  of  descrip¬ 
tion  exist,  it  is  more  difficult  for  people  to  understand  and  envision  implications  of  such  descrip¬ 
tions  until  a  final  product  has  been  developed. 
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The  goal  of  product  evaluation  is  to  encourage  practices,  regardless  of  method,  that  distinguish 
adequately  between  essential  characteristics  of  a  needed  product  and  those  that  are  only  included 
for  sufficient  descriptive  completeness  to  enable  shared  understanding.  The  best  choice  may  be  a 
mix  of  a  more  formal  notation  that  supports  precise  specifications  and  an  associated,  less  formal 
notation  that  is  more  anecdotal  and  overly  detailed  as  an  aid  to  understanding,  with  the  formal 
notation  being  authoritative. 

3.2  The  Testing  Environment,  a  Development  Effort  in  its  Own  Right 

To  provide  valid  results,  testing  requires  an  environment  that  is  an  adequate  approximation  of  the 
actual  operational  environment.  This  requires  an  operable  form  of  all  the  elements  with  which  the 
product  interacts,  including  both  the  equipment  that  comprises  the  computing  infrastructure  and 
other  systems  (adversarial  systems,  natural  forces,  unintentional  operator  actions,  etc.)  and  devic¬ 
es  that  operate  in  that  environment  and  communicate  or  share  resources  with  the  product.  If  op¬ 
erational  versions  of  other  systems  and  devices  are  not  available,  their  behavior  must  be  simu¬ 
lated. 

Just  as  the  customer’s  needs  are  the  basis  for  building  the  product,  the  basis  for  building  the  test¬ 
ing  environment  is  the  customer’s  description  of  the  product’s  operational  context.  Operational 
context  defines  the  nature  of  the  operational  environment  into  which  the  product  will  be  deployed. 
If  the  customer’s  characterization  is  not  properly  understood  and  adequately  realized  in  the  test 
environment,  evaluations  of  the  product  may  lead  to  incorrect  conclusions  regarding  the  quality 
and  acceptability  of  the  product. 

In  addition,  both  the  environment  and  the  product  must  be  configurable  and  controllable  as 
needed  to  perform  the  various  test  scenarios.  This  includes  a  mechanism  to  initialize  the  data  state 
of  the  enviromnent  and  the  product  to  represent  a  prescribed  initial  state  for  each  test  scenario. 

The  testing  environment  must  also  support  instrumentation  that  enables  the  profiling  of  internal 
behavior  and  the  collecting  of  data  about  the  product’s  behavior  that  is  needed  to  perform  quantit¬ 
ative  and  qualitative  analyses  of  the  product’s  properties. 

The  testing  environment  is  a  framework  for  evaluating  not  only  the  operable  product,  but  also  the 
documentation  and  support  that  are  needed  by  users  of  the  product.  This  documentation  must  be 
further  augmented  with  information  on  the  use  of  the  testing  environment  itself  as  a  container  in 
which  the  product  and  encompassing  system  of  interest  operate.  To  some  degree,  the  testing  envi¬ 
ronment  is  more  complex  than  that  product  or  system  because  it  must  include  observability  me¬ 
chanisms  for  controlling  the  rate  at  which  computation  and  inputs  progress  to  permit  in-progress 
examinations  and  analyses. 

Beyond  building  the  testing  environment,  there  is  the  option  of  additional  automation  for  the  use 
of  the  environment.  This  includes  the  ability  for  one-step  operation  of  scenarios  that  perform  in¬ 
itialization,  simulate  operator  inputs,  record  test  results,  and  evaluate  results  against  expectations. 
This  is  particularly  useful  for  purposes  of  repeated  regression  testing  as  parts  of  the  product 
evolve  or  to  simulate  and  evaluate  the  product’s  behavior  in  response  to  potential  future  changes 
in  its  environment. 

In  the  same  way  that  the  product  evolves  over  its  lifetime,  the  testing  enviromnent  needs  to  ac¬ 
commodate  changes  in  its  representation  of  the  operational  environment.  If  the  testing  environ- 
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ment  is  built  in  a  way  that  anticipates  or  is  adaptable  to  future  changes,  it  should  be  usable  over 
the  full  lifetime  of  the  product  without  significant  additional  investment.  In  fact,  the  testing  envi¬ 
ronment  should  be  viewed  as  an  element  of  the  product  that  is  sustained  in  consonance  with  the 
product. 
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4  Evaluating  a  Product  during  Development 


During  product  development,  product  evaluation  is  a  collaboration  between  the  developer  and  the 
acquirer.  Product  evaluation  during  development  is  meant  to  ensure  that  the  resulting  product  will 
satisfy  the  customer’s  formal  acceptance  criteria  with  a  minimum  of  evaluation  effort  during  ac¬ 
ceptance  and  without  the  need  for  rework. 

4.1  The  Product  Development  Process 

Figure  2  depicts  a  generalized  product  development  process.  This  process  involves  repeated  feed¬ 
back  and  iteration  among  the  activities  of  development.  Arrows  indicate  infonnation  flow  be¬ 
tween  activities;  activities  may  be  performed  simultaneously  on  different  elements,  increments,  or 
versions  of  a  developing  product.  This  process  is  controlled  by  a  continuous  management  activity 
that  consists  of  planning  and  resource  allocation,  monitoring  and  metrics,  configuration  manage¬ 
ment,  and  process  quality  control.  The  nature  of  product  evaluation  during  this  process,  including 
reviews  and  testing,  is  discussed  in  more  detail  below. 


Requirements 


\ 


Figure  2:  A  Product  Development  Process 

Although  convention  seems  to  favor  distinguishing  between  activities  of  systems  and  software 
engineering,  this  discussion  takes  the  view  that  software  engineering  activities  are  properly  an 
aspect  of  systems  engineering  and  cannot  be  deferred  and  treated  separately  [Campbell  2004], 
Systems  and  software  engineering  alternatives  and  tradeoffs  are  interdependent;  implications  are 
often  systemic,  transcending  any  apparent  distinctions.  Differences  in  how  hardware  and  software 
components  are  produced  are  the  purview  of  the  implementation  activity;  this  facilitates  deferring 
or  changing  the  hardware/software  implementation  of  any  component.  This  means  that  other  ac¬ 
tivities  must  address  both  system  and  software  perspectives  within  the  systems  context:  software 
requirements,  design,  and  integration  must  be  understood  and  addressed  as  an  integral  part  of  de¬ 
fining  system  requirements,  design,  and  integration. 

Product  development  begins  with  an  analysis  of  acquirer-specified  customer  needs,  resulting  in 
the  definition  of  acceptance  criteria.  These  criteria  are  the  formal  basis  against  which  product  ac¬ 
ceptance  will  be  determined. 
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From  these  needs  and  criteria — augmented  by  engineering  analyses  of  alternatives  and  tra¬ 
deoffs — a  specification  of  the  product’s  requirements  are  derived.  Requirements  define  the  prod¬ 
uct’s  build-to  observable  behavior.  As  the  product  is  developed  (and  later  evolved),  these  re¬ 
quirements  constitute  an  as-built  specification  of  the  product’s  expected  behavior. 

Verification  that  the  product’s  requirements  comply  with  acceptance  criteria — along  with  subse¬ 
quent  verification  that  the  product  as  built  satisfies  those  requirements — should  be  sufficient  to 
provide  confidence  that  the  product  is  going  to  satisfy  product  acceptance  evaluations. 

Product  design  specifies  an  architecture  for  the  product,  defining  a  consistent  set  of  alternative 
views  of  the  product’s  structure.  These  views  specify  the  content,  interdependencies,  and  interac¬ 
tions  of  the  components  that  concretely  implement  the  product.  Further,  each  of  these  views  con¬ 
stitutes  a  partial  model  of  the  product  that  provides  answers  to  specific  questions  about  how  the 
product,  and  each  of  its  components,  is  to  be  constructed.  The  following  is  one  possible  set  of 
views: 

•  a  module  decomposition  view  that  identifies  and  specifies  the  product’s  components 

•  a  concurrency  view  that  specifies  the  process/task  scheduling  and  communications  elements 
and  interactions  implemented  by  components 

•  a  configuration  view  that  specifies  mapping  of  elements  onto  physical  devices  and  computa¬ 
tional  hardware 

•  a  dependency  view  that  defines  data  and  control  dependencies  among  components  and  is  used 
as  a  guide  to  the  composition  of  operable  product  versions  for  incremental,  reduced-capability 
integration 

The  design  must  be  reviewed  and  analyzed  to  ensure  that  it  documents  and  substantiates  a  techni¬ 
cally  credible  solution  approach  that  can  be  reasonably  implemented  with  available  time,  exper¬ 
tise,  and  resources.  The  module  decomposition  view  partitions  the  implementation  effort  into  dis¬ 
tinct  components  that  can  be  assigned  to  developers  based  on  their  having  requisite  knowledge 
and  expertise.  All  of  the  views  defined  in  the  design  must  be  realized  in  the  implementation  of 
these  components. 

For  each  of  the  components  specified  in  the  design,  an  implementation  is  designed,  written,  and 
verified  in  keeping  with  the  various  design  views.  Implemented  components  should  be  verified 
through  active  reviews.  Component-level  testing  can  be  used  to  analyze  whether  each  component 
provides  expected  interface  capabilities,  behavior,  and  effects  on  quality  factors.  Testing  can  inte¬ 
grate  already-verified  dependent  and  supporting  components  to  reduce  the  test  preparation  effort. 

Integration,  as  the  complement  of  design,  creates  an  integrated  operable  version  of  the  product  by 
configuring  and  composing  a  selected  set  of  verified  components,  as  specified  in  the  dependency 
view  of  the  product  architecture.  This  product  version  (which  may  be  incomplete)  is  then  verified 
against  anticipated  product  behavior  as  specified  by  the  requirements.  The  integrated  product 
should  be  instrumented  for  use  in  the  testing  environment  to  enable  dynamic  monitoring,  profil¬ 
ing,  and  analysis  of  its  operational  behavior.  The  product  may  be  integrated  with  operational 
hardware  and  installed  into  its  (actual  or  simulated)  operational  enviromnent  for  more  realistic, 
but  controlled  experimental  use  of  the  product  following  simplified  and  actual  customer  usage 
scenarios. 
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Product  evaluation,  as  an  element  of  integration,  verifies  that  the  product  as  built  and  operated 
within  the  development  environment  satisfies  its  requirements.  If  the  requirements  were  verified 
as  being  responsive  to  customer  acceptance  criteria,  it  is  reasonable  to  expect  that  the  product 
should  in  turn  satisfy  the  less  restrictive  customer  acceptance  criteria  and  therefore  be  ready  for 
product  acceptance.  Defects  that  arise  after  product  verification  should  be  traceable  predominantly 
to  limitations  in  being  able  to  replicate  the  system  operational  environment  within  the  develop¬ 
ment  environment. 

4.2  The  Role  of  Product  Evaluation  in  Product  Development 

During  product  development,  the  role  of  product  evaluation  is  to  ensure  that  the  completed  prod¬ 
uct  will  satisfy  the  acquirer’s  and  customer’s  acceptance  criteria.  Product  evaluation  therefore 
must  participate  in  every  product  development  activity. 

In  general,  product  evaluation’s  role  is  to  determine  whether 

•  information  is  presented  in  an  understandable  (clear,  concise,  consistent,  well-structured) 
form 

•  terminology  is  properly  defined  and  consistently  used 

•  alternatives,  tradeoffs,  and  rationale  regarding  important  decisions  are  properly  documented 
.  uncertainties,  differing  views,  and  potential  future  changes  in  needs,  technology,  or  business 

context  have  been  properly  identified  and  addressed 

For  requirements,  product  evaluation’s  role  is  to  determine  whether 

•  customer  needs  have  been  adequately  communicated  by  the  acquirer,  accurately  documented 
in  the  acceptance  criteria,  and  properly  understood  by  the  developers  as  expressed  in  the  re¬ 
quirements 

•  requirements  adequately  define  expected  observable  behavior  for  a  product  that,  properly 
built,  will  satisfy  specified  acceptance  criteria 

•  requirements  specify  that  the  product  will  include  mechanisms  and  documentation  of  ratio¬ 
nale  as  needed  to  make  effective  reviews  and  testing  feasible 

•  changes  are  being  made  in  requirements  as  development  improves  understanding  of  needs, 
alternatives,  and  tradeoffs  to  accurately  represent  the  product  being  built 

For  design,  product  evaluation’s  role  is  to  determine  whether 

•  all  aspects  of  requirements  content  are  understood  as  communicated  and  represented  ade¬ 
quately  in  the  design 

•  different  architectural  views  and  models  are  defined  as  needed  to  permit  analyses  and  evalua¬ 
tions  of  product  behavior 

•  component  specifications,  including  interfaces  and  dependencies,  provide  sufficient  guidance 
to  enable  implementation  by  developers 

•  all  significant  quality  factors  have  been  identified  and  characterized  in  terms  of  design  alter¬ 
natives  and  tradeoffs 
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•  the  design  provides  means  and  mechanisms  to  operate  properly  in  response  to  invalid  actions 
or  information,  failures,  and  faults 

For  implementation,  product  evaluation’s  role  is  to  determine  whether 

•  the  implications  and  realization  of  all  architectural  views  for  each  component  are  understood 
by  its  implementer 

•  specifications  of  related  components  are  being  properly  applied 

•  the  internal  design  of  each  component  identifies  and  properly  organizes  all  of  its  elements 

•  each  component  properly  implements  its  specification 

•  component  verification  criteria  reflect  the  ways  the  component  will  be  used 

For  integration,  product  evaluation’s  role  is  to  detennine  whether 

•  the  composition  of  a  set  of  components,  as  specified  in  the  design,  is  operable  in  the  devel¬ 
opment  test  environment 

•  representative  scenarios  for  expected  use  of  the  product  behave  as  specified  in  the  require¬ 
ments 

Throughout  development,  as  well  as  during  acceptance  and  sustainment,  product  evaluation  must 
ensure  that  when  a  defect  is  discovered  it  is  corrected  and  its  cause  is  determined.  Requisite  cor¬ 
rections  must  then  be  propagated  throughout  all  product  representations  to  maintain  consistency 
(e.g.,  correcting  a  design  defect  may  also  require  changes  in  both  requirements  and  implementa¬ 
tion).  Beyond  this  direct  response,  it  should  be  determined  whether  an  improvement  in  develop¬ 
ment  or  evaluation  practices  could  preclude  such  defects  in  the  future  or  cause  them  to  be  discov¬ 
ered  earlier. 

Product  evaluation  must  determine  whether  a  product  satisfies  customer  acceptance  criteria.  Man¬ 
agement  may  decide  that  a  product  is  adequately  responsive  to  customer  needs  and  should  be 
submitted  for  product  acceptance  even  when  certain  criteria  are  not  met.  This  can  be  the  result  of 
customer  tradeoffs  in  cost  schedule  versus  capability,  better  understanding  of  actual  needs  gained 
during  development,  or  impending  changes  in  customer  needs.  This  judgment  should  result  in  the 
revision  of  the  customer  acceptance  criteria  with  the  reasons  for  this  decision  for  reference  during 
product  acceptance  and  sustainment. 
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5  Evaluating  a  Product  after  Development 


Regardless  of  whether  an  acquisition  is  considering  an  off-the-shelf  (commercial  or  publicly  ob¬ 
tainable)  product  or  a  custom-developed  product,  the  product  should  be  evaluated  in  terms  of 
whether  its  observable  behavior — augmented  by  documented  evidence  of  its  proper  construc¬ 
tion — satisfies  identified  customer  needs.  At  this  point,  the  acceptance  criteria  should  be  a  clear 
representation  of  the  customer’s  needs.  This  high  level  of  confidence  is  due  to  the  growing  expe¬ 
rience  with  the  off-the-shelf  product  or  to  product  evaluation  efforts  during  product  development. 

To  establish  acceptability  of  a  product,  product  evaluation  is  performed  progressively  to  indepen¬ 
dently  verify  and  validate  the  product: 

•  System  verification:  During  development,  the  product  has  been  verified,  in  a  facsimile  of  the 
operational  enviromnent  under  representative  operational  scenarios,  as  meeting  its  require¬ 
ments,  and  the  requirements  have  been  validated  as  consistent  with  customer  acceptance  cri¬ 
teria.  The  product  must  be  integrated  with  other  elements  of  the  customer’s  operational  sys¬ 
tem  and  verified  as  meeting  aggregate  customer  acceptance  criteria  in  a  close  approximation 
of  the  actual  operational  environment. 

•  Operational  evaluation:  having  satisfied  system-level  product  evaluation  criteria,  the  product 
is  subjected  to  validation  involving  representative  operators/users  in  an  accurate  approxima¬ 
tion  of  the  operational  environment  and  usage  scenarios  to  detennine  if  it  is  fit  for  its  in¬ 
tended  use. 

•  Delivery:  A  validated  operational  product  must  be  installed  with  documentation,  training, 
and  assistance  for  users.  Use  of  the  product  results  in  customer  feedback  on  issues  encoun¬ 
tered,  required  workarounds,  and  projected  future  needs. 

•  Sustainment:  An  installed,  in-use  product  over  time  begins  to  encounter  changing  customer 
needs.  This  leads  to  recurring  performance  of  acquisition  activities  to  produce,  accept,  and 
deliver  revised  versions  of  the  product. 

Finding  defects  either  in  product  quality  or  in  product  fit  to  the  customer’s  needs  during  product 
acceptance  is  costly,  resulting  in  either  a  delay  for  rework  or  the  acceptance  of  an  inferior  product. 
However,  when  a  defect  that  prompts  rework  is  discovered  at  this  stage,  action  should  be  initiated 
not  only  to  correct  the  defect  through  the  required  rework  (depending  on  severity,  immediately  or 
in  a  future  product  revision),  but  also  to  determine  why  the  defect  was  not  discovered  sooner,  pre¬ 
ferably  during  product  development,  and  corrected  then.  A  subsequent  action  should  be  to  amend 
product  evaluation  procedures  so  that  this  and  any  similar  defects  are  detected  during  develop¬ 
ment,  rather  than  during  product  acceptance. 

5.1  The  Role  of  Product  Evaluation  in  Determining  Product  Acceptance 

Optimistically  assuming  that  there  are  no  defects,  a  product  should  conform  to  its  associated  as- 
built  requirements.  The  requirements  should  define  what  behavior  can  be  expected  of  the  product 
in  operation.  The  purpose  of  product  evaluation  in  this  case  is  to  determine  whether  this  expected 
behavior  meets  both  customer  acceptance  criteria  and  more  broadly,  current  actual  customer 
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needs.  If  the  product  fails  to  adequately  satisfy  this  criteria,  the  acquirer  and  customer  may  decide 
to  reject  the  product,  accept  the  product,  or  consider  critical  changes  to  make  the  product  accepta¬ 
ble.  Changing  a  product  that  failed  to  satisfy  its  acceptance  criteria  requires  making  corresponding 
revisions  in  that  criteria  so  that  it  matches.  The  evaluation  of  requirements  during  product  devel¬ 
opment  should  have  discovered  any  discrepancies  with  acceptance  criteria;  any  failure  to  accept  at 
this  point  would  be  the  result  of  not  having  properly  expressed  the  customer’s  needs  or  by  an  un¬ 
anticipated  change  in  the  customer’s  needs. 

Although  requirements  provide  a  nominal  basis  for  evaluating  the  product,  actual  acceptance  re¬ 
lies  on  criteria  that  may  be  implied  in  the  requirements,  but  are  made  explicit  only  in  the  form  of 
evaluation  criteria.  These  criteria  are  created  specifically  as  a  guide  for  product  evaluation,  to  de¬ 
fine  more  precisely  what  requirements  are  understood  to  mean  operationally. 

A  product  is  judged  acceptable  to  the  degree  that  it  behaves  in  the  way  that  the  customer  expects, 
regardless  of  whether  the  expected  behavior  was  accurately  communicated  in  the  requirements. 
Product  evaluation,  as  a  proxy  for  acquirer  and  indirectly  customer  judgment,  must  establish 
through  its  realization  of  acceptance  criteria  whether  a  product  is  behaviorally  acceptable  for  dep¬ 
loyment  and  use. 

5.2  Product  Evaluation  for  the  Purpose  of  Product  Acceptance 

When  product  development  results  in  a  candidate  product  release,  the  acquirer  must  determine 
whether  the  product  as  built  is  acceptable  for  deployment  into  operation. 

Product  acceptance  includes  three  levels  of  product  evaluation: 

•  Verification:  determining  that  the  product  is  a  correct  solution  to  the  problem  as  understood 
and  expressed  in  customer  acceptance  criteria. 

•  Validation:  determining  that  the  product  properly  addresses  the  customer’s  actual  current  and 
future  needs. 

•  Certification:  determining  that  the  product  satisfies  all  organizational  and  statutory  require¬ 
ments  concerning  safety,  security,  and  legal  regulation  applicable  to  its  operational  environ¬ 
ment. 

Verification  during  product  acceptance  consists  of  reviews  of  documents  for  consistency  and 
completeness  and  system  testing.  This  is  a  determination  that  the  product  has  been  built  as  speci¬ 
fied  and  works  properly  with  other  separately  provided  components  of  the  system.  If  product 
evaluation  was  properly  performed  during  development,  failures  of  this  type  should  not  occur  dur¬ 
ing  acceptance;  any  verification  defect  found  during  product  acceptance  should  be  traced  to  its 
source  in  product  development  so  that  the  product  evaluation  processes  can  be  refined  to  expose 
this  type  of  defect  earlier. 

Validation  during  product  acceptance  takes  the  form  of  operational/acceptance  testing.  This  is  the 
determination  that  the  product  provides  the  customer  with  the  capabilities  needed  to  perform  their 
work  effectively;  this  means  that  the  requested  capabilities  correspond  to  what  the  customer  ac¬ 
tually  needs  at  the  time  of  completion.  The  evaluation  can  fail  if  the  acquirer  misunderstood  the 
customer’s  needs  or  specified  those  needs  incorrectly;  this  type  of  failure  is  less  likely  if  repre¬ 
sentative  users  have  been  consulted  during  product  development.  Alternatively,  this  evaluation 
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can  fail  if  the  developer  misunderstood  the  customer’s  needs  as  communicated  by  the  acquirer 
This  misunderstanding  should  not  be  discovered  during  product  acceptance,  if  during  product  de¬ 
velopment,  the  proper  use  of  requirement  and  design  reviews  were  conducted  reflecting  the  cus¬ 
tomer’s  needs. 

Validation  requires  the  formal  participation  of  customer  representatives  and  is  not  constrained  by 
formal  acceptance  criteria  defined  for  the  acquisition.  The  success  of  validation  depends,  howev¬ 
er,  on  the  acceptance  criteria  having  accurately  characterized  the  acquirer’s  and  customer’s  expec¬ 
tations  concerning  the  needed  product’s  capabilities  and  behavior.  If  those  needs  have  been  mi¬ 
sunderstood  or  misrepresented,  or  have  subsequently  changed,  the  product  may  be  deemed 
unacceptable  by  the  customer.  The  role  of  product  evaluation  in  this  respect  is  to  establish  and 
maintain  visibility  with  the  acquirer  and  customer  representatives — and  their  active  participation 
when  possible — to  ensure  that  the  acceptance  criteria  properly  describe  their  expectations.  Accep¬ 
tance  criteria  should  include  not  only  a  characterization  of  customer  needs,  as  they  initially  exist, 
but  also  a  characterization  of  uncertainties  and  potential/anticipated  changes  in  those  needs. 
Awareness  of  uncertainties  and  prospective  changes  during  development  facilitates  modifications 
in  the  product  both  before  deployment  and  during  sustainment. 

Certification  is  the  determination  that  the  product  conforms  to  organization,  industry,  and  gov¬ 
ernment  rules  and  regulations  regarding  safety  and  security.  The  role  of  product  evaluation  in  this 
process  is  to  ensure  that  the  product  has  been  built  in  such  a  fashion  that  it  will  meet  these  criteria. 
This  is  substantiated  by  the  proper  recording,  as  part  of  process  quality  assurance,  of  associated 
safety  and  security  assurance  actions  taken  during  development.  As  a  rule,  these  criteria  will  be 
met  if  the  product  development  process  can  be  seen  to  have  properly  addressed  the  issues  and  tra¬ 
deoffs  that  motivate  standards  of  safety  and  security. 

5.3  Product  Evaluation  as  an  Aspect  of  Sustainment 

For  most  of  their  useful  lives,  products  are  in  sustainment,  encompassing  both  maintenance  and 
follow-on  acquisition.  Maintenance  includes  monitoring  and  adjusting  operational  parameters, 
detecting  and  correcting  residual  defects,  and  replacing  expendable  parts  and  supplies,  whereas 
follow-on  acquisition  involves  modifying  or  replacing  the  product  over  time  as  the  customer’s 
needs  change  so  that  those  needs  will  continue  to  be  satisfied.  As  the  proper  complement  to  de¬ 
velopment,  follow-on  acquisition  requires  that  all  aspects  of  development  and  acceptance  be  re¬ 
peated  in  order  to  ensure  not  only  needed  advances,  but  also  continued  integrity  of  the  product’s 
specification  and  behavior.  Only  by  revisiting  all  aspects  of  development  and  acceptance  can  the 
acquirer  and  customer  be  assured  that  all  implications  of  the  required  changes  have  been  properly 
identified  and  addressed.  Similarly,  in  the  context  of  sustaimnent,  all  elements  of  product  evalua¬ 
tion  must  be  revisited  to  ensure  proper  repeat  performance  of  an  acquisition’s  product  develop¬ 
ment  and  acceptance  activities. 

It  is  not  sufficient  during  follow-on  acquisition  to  consider  only  the  direct  effects  of  changes  on 
the  product — specifically  for  software  but  also  to  a  degree  for  hardware.  Implications  of  a  change 
can  often  cascade  into  unexpected  areas  of  behavior  through  complex  low-level  interactions  such 
as  indirect  sharing  of  resources  or  unpredictable  sequencing  of  communications  and  processing. 

In  some  cases,  hardware  can  be  sustained  by  substitution  of  one  part  with  another  of  physically 
equivalent  but  improved  capability;  even  in  this  case,  as  with  software  changes,  improved  capabil- 
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ity  usually  implies  potential  behavioral  differences  that  may  change  timing  or  informational  de¬ 
pendencies  with  other  parts  of  a  system,  requiring  that  the  system  be  reevaluated. 

As  a  sustainment  activity,  the  details  of  product  evaluation  differ  from  prior  occurrences  to  the 
degree  that  the  product  itself  differs.  There  may  be  changes  in  requirements,  design,  implementa¬ 
tion,  and  integration  work  products  that  require  corresponding  changes  in  the  materials  of  product 
evaluation.  For  example,  changes  may  mean  that  different  quality  factor  tradeoffs  would  need  to 
be  considered  or  different  user  scenarios  would  need  to  be  tested.  Similarly,  new,  modified,  or 
deleted  components  that  integrate  differently  could  result  in  new  or  changed  product  behavior  and 
would  require  testing  revisions.  The  infrastructure  of  product  evaluation  developed  during  the 
initial  acquisition  can  still  be  used,  but  its  details  must  be  revised  to  account  for  the  implications 
of  any  changes.  During  iterations  of  the  initial  acquisition,  the  product  must  be  reevaluated  to  en¬ 
sure  that  prior  behavior  remains  unchanged  with  respect  to  any  aspects  that  should  not  have  been 
affected  by  the  changes  (i.e.,  regression  reviews  and  testing). 
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6  Product  Evaluation  in  a  Product  Line  Context 


The  traditional,  large  system  conception  of  acquisition  is  the  development  of  a  manufacturing  fa¬ 
cility  that  is  operated  to  produce  a  series  of  largely  identical  products.  More  recently,  the  manu¬ 
facturing  approach  has  evolved  to  permit  the  products  coming  off  the  line  to  differ  in  ways  that 
are  more  substantial.  With  the  increasing  reliance  on  software  as  a  determinant  of  product  capabil¬ 
ity,  the  opportunity  exists  to  customize  each  product  to  satisfy  specific  operational  needs.  From  a 
systems  and  software  engineering  perspective,  this  opportunity  has  been  fonnalized  as  a  systemat¬ 
ic  “product  line”  approach  to  acquisition  [Campbell  2002]. 

6.1  The  Concept  of  a  Product  Line 

A  product  line  is  a  set  of  products  that  satisfy  similar  needs.  Differences  in  similar  products  may 
be  essential — corresponding  to  differences  in  customers’  needs — or  they  may  be  incidental,  cor¬ 
responding  to  differences  in  how  developers  have  chosen  to  build  the  products. 

Although  similar  products  can  and  have  been  developed  independently,  doing  so  entails  duplicate 
efforts  during  both  development  and  sustainment  and  typically  leads  to  unnecessary  differences 
among  those  products.  This  duplication  can  be  somewhat  mitigated  when  the  same  engineers  are 
responsible  for  those  products,  to  the  degree  that  they  recognize  and  can  maintain  the  similarity  of 
the  products  over  their  lifetimes;  however,  a  more  systematic  approach  exists  for  creating  cohe¬ 
rent  product  lines.  This  approach — based  on  the  concept  of  a  product  family — optimizes  produc¬ 
tivity  and  quality  by  minimizing  essential  differences  and  avoiding  incidental  differences  among 
products. 

A  product  family  envisions  a  potential  set  of  similar  products;  these  products  are  alike  in  many 
ways  (commonalities)  but  differ  in  certain  particular  ways  (variabilities).  A  product  family 
represents  a  set  of  products  implicitly,  in  an  abstract  form  that  provides  the  means  to  rapidly  de¬ 
rive  a  product  that  is  customized  to  fit  a  customer’s  specific  needs.  Variabilities  represent  custom¬ 
er  and  engineering  choices — in  areas  such  as  mission,  business  practices,  technology,  and  opera¬ 
tional  environment — that  must  be  made  in  order  to  build  a  specific  product.  These  choices  (in 
aggregate)  determine  the  exact  content  of  each  work  product,  from  requirements  and  design  speci¬ 
fications  to  component  implementations  and  user  documentation.  The  products  that  are  derived  in 
this  way,  as  instances  of  a  product  family,  constitute  a  coherent  product  line.  The  effort  to  build 
individual  products  leverages  the  effort  applied  to  creating  the  product  family,  resulting  in  the 
elimination  of  duplicate  efforts  across  the  product  line. 

An  analogous  case  exists  for  treating  a  single  product  as  a  product  line  opportunity.  For  most 
products,  there  are  uncertainties  in  customer  needs  and  expectations  for  potential  changes  across 
the  useful  life  of  the  product.  These  uncertainties  and  changes  can  be  accommodated  by  conceiv¬ 
ing  of  the  evolving  versions  of  the  product  as  being  distinct  instances  of  an  encompassing  product 
line:  each  new  version  of  the  product  could  then  be  derived  as  an  instance  of  that  product  line. 

As  discussed  more  broadly  in  McGregor,  a  product  line  approach  suggests  two  opportunities  for 
improving  product  evaluation  costs  and  effectiveness.  The  first  opportunity,  exploiting  the  as¬ 
sumed  similarity  of  products,  leverages  evaluation  efforts  across  the  product  line  by  pre-verifying 
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product  line  assets;  this  will  improve  the  quality  of  each  individual  product  while  reducing  the 
evaluation  effort.  The  second  opportunity  concerns  applying  product  line  concepts  in  the  devel¬ 
opment  of  product  testing  assets  and  capabilities  [McGregor  2001]. 

6.2  Evaluating  a  Product  Line  to  Reduce  Product  Evaluation  Efforts 

The  first  opportunity  for  improvement  results  directly  from  the  need  to  build  similar  products.  By 
specifying  an  architecture  that  describes  any  of  the  family  of  products  equally  well,  a  product  line 
defines  a  set  of  components  that  can  be  reused  to  build  customized  instances  of  that  product  fami¬ 
ly.  As  with  developing  any  component,  assets  should  be  separately  evaluated  prior  to  use  in  any 
product.  These  reviews  and  tests  will  provide  an  initial  level  of  assurance  that  the  components 
will  be  usable  without  further  effort  in  any  instance  of  the  product  family;  in  addition,  because 
these  assets  are  used  in  multiple  products,  subsequent  testing  of  any  instance  product  built  using 
an  asset  has  the  effect  to  some  degree  of  testing  that  asset  across  all  similar  products.  With  this 
added  basis  in  prior  testing,  not  only  will  evaluation  efforts  for  future  products  be  reduced,  but 
those  products  will  also  exhibit  higher  quality,  with  fewer  defects  requiring  less  rework. 

The  objective  with  a  product  line  is  to  substantially  reduce  the  time  required  to  create  a  product 
with  a  specified  level  of  quality  (e.g.,  no  known  critical  defects  and  meeting  threshold  levels  of  all 
quality  factors).  Although  a  product  line  does  enable  the  rapid,  low-cost  development  of  new  and 
modified  products,  the  artifacts  that  constitute  product  line  assets  should  generally  be  evaluated 
prior  to  use  against  the  same  criteria  and  using  the  same  techniques  that  would  be  used  in  evaluat¬ 
ing  a  conventionally  developed  product.  Even  so,  using  properly  evaluated  product  line  assets  can 
result  in  a  substantial  reduction  in  the  time  and  expertise  required  to  build  and  evaluate  each  sub¬ 
sequently  derived  instance  product. 

In  a  product  line  context,  evaluations  occur  at  the  component,  work  product,  and  product  levels.  A 
product  line  asset  at  any  of  these  levels  corresponds  to  a  set  of  similar  instance  assets;  therefore, 
an  evaluation  may  address  either  a  single  derived  instance  or  a  set  of  instances  represented  by  the 
asset.  Any  asset  evaluation  consists  of  a  combination  of  peer/mentor/expert  reviews,  testing,  and 
formal  methods.  Any  defect  found  in  the  specification  or  implementation  of  any  asset  instance  is 
treated  as  being  a  defect  in  the  asset  as  a  whole. 

A  single-instance  strategy  for  evaluating  an  asset  entails  (1)  identifying  relevant  instance-level 
evaluation  criteria,  (2)  deriving  an  instance  that  confonns  to  evaluation  preconditions,  (3)  evaluat¬ 
ing  the  instance  relative  to  expected  results,  and  (4)  modifying  the  asset  to  correct  for  any  defects. 
By  selecting  and  evaluating  a  variety  of  instances  of  the  asset,  the  quality  of  the  asset  as  a  whole 
will  increase,  along  with  confidence  that  any  subsequent  instance  derived  will  be  less  likely  to 
exhibit  defects.  This  strategy  is  applicable  when  an  instance  for  use  in  a  product  is  derived  and 
verified  as  part  of  the  product  evaluation  effort;  any  defects  that  are  found  in  this  way  should  be 
used  to  correct  the  asset,  not  just  the  specific  instance. 

A  refinement  of  a  single-instance  strategy  is  a  sampled-instances  strategy.  By  statistically  select¬ 
ing  instances  for  evaluation,  based  on  the  distribution  of  references  to  the  variabilities  across  the 
set,  a  higher  level  of  confidence  can  be  more  systematically  achieved  for  a  given  level  of  evalua¬ 
tion  effort.  By  selecting  instances  that  are  sufficiently  diverse  to  cover  the  set,  less  evaluation  ef¬ 
fort  is  needed  to  discover  most  asset  defects. 
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The  greatest  leverage  is  provided  by  an  abstract  strategy  for  evaluating  an  asset.  With  this  strate¬ 
gy,  the  abstract  representation  of  the  set  of  instances  is  directly  analyzed.  This  requires  using  a 
method  that  permits  reasoning  logically  over  an  abstractly  characterized  set.  When  using  a  formal 
method  or  model  that  was  designed  for  verifying  instance  properties,  variabilities  correspond  to 
universal  quantifiers  that  extend  the  solution  space  over  which  the  properties  can  be  evaluated. 
This  approach  provides  the  means  to  demonstrate  that  a  property  holds — not  just  for  individual 
asset  instances — but  also  for  every  instance  in  the  set.  In  addition,  the  conformance  of  the  instance 
set  to  its  specification  can  be  reasoned  over  the  logical  space  determined  by  the  variabilities  that 
characterize  the  asset  as  being  a  set. 

In  lieu  of  a  formal  method,  evaluators  can  use  review  or  static  testing  to  analyze  whether  any 
combination  of  variability  choices  would  result  in  an  instance  that  would  violate  an  asset’s  cor¬ 
rectness  properties.  These  informal  techniques  are  imperfect  but  can  increase  confidence  that  all 
instances  of  the  asset  satisfy  critical  criteria  without  having  to  separately  evaluate  every  derivable 
instance  of  the  asset.  These  properties  should  still  be  verified  specifically  for  each  product  that  is 
subsequently  derived  for  a  customer;  however,  the  effort  to  evaluate  a  derived  product  will  be 
significantly  reduced  due  to  the  improved  quality,  absence  of  defects,  and  reduced  need  for  re¬ 
work  resulting  from  prior  family-level  evaluations. 

6.3  Establishing  a  Testing  Product  Line 

The  second  opportunity  for  improvement  results  from  needing  to  repeatedly  test  multiple  similar 
products  or  versions.  Because  products  are  similar,  the  required  testing  is  also  similar — requiring 
similar  test  plans,  test  scenarios  and  cases,  test  data,  and  test  results — alike  to  the  degree  that  cus¬ 
tomers’  needs  are  alike  and  differing  to  the  degree  that  they  differ.  By  developing  testing  mate¬ 
rials  that  are  customizable  in  terms  of  how  products’  behaviors  are  meant  to  differ  (i.e.,  treating 
them  as  product  line  assets),  redundant  preparations  for  separately  testing  each  product  can  be 
eliminated.  Investing  in  testing  product  line  assets  can  pay  off  even  if  the  tested  products  are  be¬ 
ing  developed  conventionally,  as  long  as  the  similarity  in  their  behaviors  is  understood. 

Just  as  product  line  variabilities  characterize  customer  and  engineering  choices  that  determine  the 
exact  content  of  each  product,  similar  choices  determine  the  content  of  each  product’s  test  mate¬ 
rials.  Rather  than  building  test  materials  uniquely  for  each  product  or  building  them  for  one  prod¬ 
uct  and  then  modifying  them  uniquely  for  each  succeeding  product,  product  line  testers  would 
instead  create  adaptable  test  materials.  Adaptable  materials  are  explicitly  customized,  structured, 
and  annotated  based  on  the  variabilities  associated  with  the  product  line. 

A  further  opportunity  to  leverage  product  line  concepts  for  testing  is  to  create  an  adaptable  testing 
environment.  With  adaptability,  the  testing  environment  can  be  configured  as  needed  to  accurate¬ 
ly  replicate  a  variety  of  targeted  business  operational  enviromnents.  This  is  beneficial  when  build¬ 
ing  similar  products  for  different  customers  or  for  a  customer  whose  operational  environment  can 
vary  geographically  or  over  time.  In  these  cases,  each  product  needs  to  be  evaluated  in  a  facsimile 
operational  environment  that  has  been  tailored  to  match  the  customer’s  specified  environment. 

The  more  sensitive  a  product’s  behavior  is  to  its  operational  environment,  the  more  important  it  is 
to  evaluate  the  product  in  a  test  environment  that  accurately  mimics  the  behavior  of  the  actual 
environment.  Again,  rather  than  building  or  modifying  a  facsimile  operational  environment  for 
every  product,  the  facsimile  enviromnent  can  be  constructed  as  an  instance  of  a  family  that  is  to 
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be  adaptable  to  the  variabilities  that  characterize  the  operational  environments  into  which  prod¬ 
ucts  may  be  deployed.  This  can  have  the  added  advantage  that  the  product  will  need  less  extensive 
testing  in  its  actual  operational  environment,  where  the  cost  of  testing  as  well  as  the  risks  and  cost 
of  correcting  defects  tend  to  be  excessive. 
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7  Improving  Product  Evaluation  Practices 


Opportunities  exist  in  most  software  development  efforts  to  improve  and  streamline  product  eval¬ 
uation  practices.  The  preceding  sections  have  provided  some  perspectives  in  general  on  how 
product  evaluation  can  be  performed  most  effectively  within  an  acquisition  effort.  These  perspec¬ 
tives  were  based  on  specific  anecdotal  experiences  in  working  with  a  variety  of  DoD  acquisition 
programs  and  their  development  contractors.  Listed  below  are  examples  of  limited  propositions 
that  may  (in  appropriate  situations)  result  in  beneficial  improvements: 

Establish  precise,  objective  criteria  for  product  acceptance 

The  role  of  product  evaluation  in  product  acceptance  is  to  define  a  set  of  usage  scenarios  that  will 
expose  whether  a  product’s  behavior  provides  the  capabilities  expressed  in  customer  needs.  The 
basis  for  these  scenarios  should  be  acceptance  criteria  that  formalize  the  acquirer’s  understanding 
of  customer  needs.  Product  acceptance  should  be  defined  as  demonstrating  that  a  product  behaves 
as  expected  for  a  set  of  scenarios  that  properly  represent  the  specified  acceptance  criteria.  Any 
product  that  satisfies  product  requirements,  which  in  turn  has  been  verified  as  being  in  confor¬ 
mance  with  acceptance  criteria,  should  operate  properly  under  usage  scenarios  that  are  believed  to 
be  representative  of  criteria. 

The  goal  of  product  development  is  to  provide  such  a  product.  In  addition,  product  quality  should 
be  measured  in  terms  of  the  effectiveness  of  design  tradeoffs  in  resolving  conflicting  quality 
attributes.  Based  on  statistical  analyses  of  product  evaluation  efforts  prior  to  initiating  product 
acceptance,  the  product  should  meet  an  expected  level  of  confidence  that  it  is  free  of  unknown 
residual  defects. 

Designate  product  evaluators  as  authoritative  proxies  for  the  customer  and 
associated  users 

The  purpose  of  product  evaluation  is  to  ensure  that  the  product  as-built  is  the  product  that  best 
meets  the  customer’s  actual  needs.  To  achieve  this,  there  must  be  a  constant  customer  “presence” 
throughout  development.  Product  evaluators  may  themselves  be  users  but  more  typically  will  be 
people  who  broadly  understand  essential  aspects  of  the  customer  enterprise,  including  users’ 
roles,  activities,  and  goals.  This  expertise  should  be  augmented  through  direct  liaison  with  other 
domain  experts  and  knowledgeable  users.  Fixing  a  product  that  is  defective  but  essentially  fits  the 
customer’s  needs  is  much  easier  than  correcting  a  soundly  built  but  conceptually  flawed  product. 

Pair  product  evaluators  with  developers  throughout  product  development 

Product  evaluators  are  likely  to  have  a  better  understanding  of  what  the  customer  needs  as  product 
developers  work  through  alternative  solutions.  Product  evaluators  gain  deeper  insight  into  solu¬ 
tion  tradeoffs  as  product  developers  gain  deeper  insight  into  customer  needs.  In  the  case  of  com¬ 
ponent  implementers,  product  evaluators  will  provide  expertise  and  access  to  facilities  that  can 
reduce  the  developer’s  unit  testing  efforts  and  ensure  readiness  for  integration  testing.  With  ap¬ 
propriate  expertise,  a  product  evaluator  can  pair  with  developers  of  multiple  components  to  gain 
improved  design-level  insights;  similarly,  developers  who  are  familiar  with  the  customer’s  needs 
can  be  product  evaluators  for  other  developers. 
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Perform  product  evaluation  activities  throughout  product  development 


The  lowest  cost  time  to  correct  defects  in  a  product  is  early,  while  the  product  is  being  developed. 
Reviews,  model-based  analyses,  or  tests,  as  appropriate  to  particular  work  products,  should  be 
performed  continuously  with  a  focus  particularly  on  areas  of  greatest  risk  as  judged  by  the  devel¬ 
oper.  Requirements  and  design  reviews  should  focus  particularly  on  establishing  bounds  for  sys¬ 
temic  quality  factors  (e.g.,  performance,  reliability,  safety,  security,  usability)  and  how  the  prod¬ 
uct  will  satisfy  these.  Similarly,  implementation  reviews  should  focus  on  identifying  and  handling 
abnormal  conditions. 

Institute  recurring  active  reviews  of  every  work  product  as  it  is  being  developed 

Focused  peer/mentor/expert  reviews  are  the  earliest,  least  expensive,  and  most  comprehensive 
means  of  finding  defects  in  a  product;  active  reviews  are  an  effective  method  of  perfonning  such 
reviews.  Reviewers  are  selected  for  specific  expertise  and  focused  on  answering  specific  topical 
questions  about  aspects  of  the  work  product  that  the  developer  feels  least  certain  of.  An  assigned 
product  evaluator  is  a  reviewer  of  first  resort,  with  a  focus  on  consistency,  quality,  and  fit  to  cus¬ 
tomer  needs.  Reviews  by  authors  of  related  development  work  products  can  expose  inconsisten¬ 
cies  and  misunderstandings:  in  particular,  requirements  authors  and  designers  should  continuously 
review  implementation  work  products  to  ensure  developers  have  a  shared  understanding  [Pamas 
1985], 

Substitute  focused  expert  technical  reviews  for  formal  management  progress 
reviews  and  technical  oversight  meetings 

Management  meetings  typically  consume  too  much  time  of  too  many  people  who  should  be  doing 
other  work.  Automate  tracking  and  reporting  of  planned  progress,  with  management  meetings 
scheduled  “by  exception”  when  advances  or  slips  in  schedules  warrant  re-planning  or  resolution 
of  specific  coordination  issues.  Use  meetings  as  small,  topically  focused  venues  for  resolving  re¬ 
source  utilization  conflicts  or  for  identifying  and  resolving  difficult  requirements  and  engineering 
tradeoffs  with  responsible  developers.  Include  knowledgeable  evaluators  and  subject  matter  ex¬ 
perts  in  these  meetings. 

Institute  rapid  iteration  over  the  product  development  process 

The  best  critics  of  a  work  product  are  other  developers  who  use  it:  designers  discover  require¬ 
ments  issues,  implementers  discover  design  issues,  and  integrators  discover  implementation  is¬ 
sues.  Looking  across  all  the  work  products,  evaluators  discover  developer  misconceptions,  mi¬ 
sunderstandings,  and  inconsistencies.  No  work  product  should  be  considered  finished  until  it  has 
been  applied  successfully,  after  correction  of  any  discovered  discrepancies.  Incremental  develop¬ 
ment  and  repeated  iterative  refinement  of  each  work  product  is  the  shortest  path  to  a  quality  prod¬ 
uct. 

Task  product  evaluators  to  verify  consistency  across  product  development 
activities 

Beyond  establishing  that  each  element  of  the  product  exhibits  a  correct  understanding  of  customer 
needs,  product  evaluation  needs  to  ensure  that  all  elements  are  defining  a  consistent  solution  ap¬ 
proach  to  meeting  those  needs. 
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Ensure  that  products  are  evaluated  for  proper  behavior  in  nominal,  stress,  and 
abnormal  conditions 

Typically,  the  focus  of  evaluation  is  on  how  the  product  is  expected  to  behave  “normally,”  provid¬ 
ing  the  capabilities  that  the  customer  needs;  however,  insufficient  focus  on  product  behavior  un¬ 
der  stress  or  abnormal  conditions  can  result  in  missing  defects  that  will  lead  to  operational  failures 
at  the  worst  times.  Stress  conditions  include  performance  and  usability  under  heavy  load  such  as 
communications  or  input  data  losses  or  reduced  responsiveness  to  user  actions.  Abnormal  condi¬ 
tions  include  device  failures,  data  integrity  losses  under  load,  security  lapses  due  to  attempted  mi¬ 
suse,  or  failures  due  to  residual  defects. 

Recognize  that  an  integration  failure  represents  a  lack  of  design-implementation 
discipline 

It  is  the  responsibility  of  the  design  to  define  the  components  that  are  implemented  to  build  the 
product  and  how  those  components  interact.  Integration  fails  only  if  the  design  is  flawed  or  a 
component  implementation  has  failed  to  conform  to  the  design.  These  failures  should  be  found  in 
design  and  implementation  evaluations,  not  when  integration  is  attempted — integration  should  be 
a  purely  mechanical  process  that  never  fails  if  implementation  evaluations  have  been  properly 
performed. 

Plan  to  discover  defects  closest  in  time  to  their  cause 

The  most  cost-effective  time  to  discover  a  defect  is  immediately  after  its  creation.  Deferral  of  ef¬ 
forts  that  could  discover  a  defect  until  later  in  the  process  only  serves  to  increase  the  difficulty  of 
finding  and  correcting  it  and  the  risk  that  rework  will  take  unplanned  time  and  introduce  new  de¬ 
fects. 

Whenever  a  defect  is  found,  evaluate  whether  the  defect  could  have  been  detected  earlier.  Identify 
product  evaluation  actions  that  would  have  detected  such  defects  and  plan  suitable  verification 
actions  to  ensure  earlier  detection  of  these  defects  in  the  future. 

Use  automation  to  reduce  product  evaluation  effort 

Product  evaluation  can  be  time  consuming  and  repetitious.  Such  work  can  be  automated,  in  part 
with  commercial  tools  and  otherwise  using  product  line  techniques.  Initial  setup  of  evaluation 
mechanisms  may  require  manual  effort  but  the  application  of  these  mechanisms  and  analysis  of 
results  against  expectations  should  be  largely  automated.  The  goal  should  be  to  require  human 
effort  only  by  exception,  such  as  when  the  product  exhibits  unexpected  behavior.  Recognize, 
however,  that  automation  of  product  evaluation  efforts  is  itself  a  development  effort,  requiring 
appropriate  expertise  in  both  software  engineering  methods  and  product  evaluation  techniques. 

Breakdown  artificial  process  boundaries 

Treat  product  evaluation,  including  testing,  not  as  a  separate  activity  but  as  an  integral  aspect  of 
every  development  activity.  No  activity  should  be  considered  complete  until  it  has  satisfied  expli¬ 
cit  evaluation  criteria,  with  the  goal  of  having  no  avoidable  defects  that  will  cause  subsequent  re¬ 
work.  Throughout  development,  developers  and  evaluators  should  collaboratively  review  and  test 
the  product  to  ensure  with  high  certainty  that  post-development  evaluation  will  not  find  any  criti- 
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cal  defects.  Developers  and  evaluators  should  also  work  together  throughout  development  to  en¬ 
sure  that  there  is  a  common  understanding  of  customer  needs  and  tradeoffs  so  that  defects  will  be 
avoided  or  found  and  corrected  early. 

Track  effort  in  terms  of  tasks  requiring  similar  skills  rather  than  by  project  phase 

Although  a  project  might,  for  management  purposes,  be  organized  into  a  development  and  accep¬ 
tance  phase,  both  phases  require  performance  of  development  and  testing  tasks.  Staffing  and 
budgets  should  reflect  that  these  skills  are  needed  in  both  phases.  Rework  that  occurs  during  the 
test  phase  should  be  viewed  as  continuing  development  work,  not  as  a  normal  aspect  of  testing. 
Similarly,  product  evaluators  that  help  developers  during  the  development  phase  to  better  test 
their  work  products  are  nevertheless  performing  test,  not  development,  activities.  Because  of  this 
functional  overlap,  improvements  in  either  development  or  test  practices  will  benefit  both  phases. 
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